Skip to content

feat(processors): index .doc/.docx, .xls/.xlsx and .msg documents - #1265

Open
cbcoutinho wants to merge 5 commits into
fix/pymupdf-table-linebreaksfrom
feat/office-msg-processors
Open

feat(processors): index .doc/.docx, .xls/.xlsx and .msg documents#1265
cbcoutinho wants to merge 5 commits into
fix/pymupdf-table-linebreaksfrom
feat/office-msg-processors

Conversation

@cbcoutinho

@cbcoutinho cbcoutinho commented Aug 6, 2026

Copy link
Copy Markdown
Owner

Adds three processors, each reading its format the way that measured best
rather than forcing one route on all of them.

Word documents (.doc/.docx) go through a LibreOffice PDF rendition parsed at
the structured tier. Legacy .doc has no working pure-Python reader at all --
markitdown raises UnsupportedFormatException and docling-serve returns
status=failure -- and for .docx the rendition is more faithful than a direct
parse, not less: 98.3% token recall with 22 table rows against mammoth's 17,
because mammoth drops a vertically-merged cell and shifts the rest of that row
one column left, landing answers under the wrong heading. The rendition is a
real PDF, so page numbers and highlight geometry come from the existing path
instead of a second implementation. Hyperlink targets are lost in exchange.

The tier is forced, not classified: the classifier scores a rendition on text
quality and returns tier='fast', and the fast tier extracts ZERO tables from
one, because rendered borders are vector line-art that only find_tables sees.

Spreadsheets (.xls/.xlsx) are read cell-by-cell and deliberately never
rendered. PDF rendering paginated a real workbook into 25 mixed-orientation
pages and recalled only 63.8% of the tokens a direct read recovers -- 567
cells against 2013 -- losing the question numbering entirely, because
LibreOffice honours the print layout. Legacy .xls converts container-to-
container via xlsx, which changes the format without re-laying it out.
Chunks carry sheet_boundaries (sheet + cell range) where a PDF carries
pages; a spreadsheet has no geometry to highlight.

Outlook messages (.msg) are read with a small olefile-based reader rather than
extract-msg, which depends on red-black-tree-mod -- published as an sdist
only, so it cannot satisfy the image's uv sync --no-build (no dependency's
setup.py may execute at build time, docker:S8541). The reader is ~80 lines and
recovers slightly more than extract-msg did on a real 1.3 MB thread (15,313
characters against 15,303) about seven times faster, while dropping nine
transitive dependencies. Reading both string variants is the substance of
it: markitdown emits 27 bytes for that message -- the literal "# Email
Message\n\n## Content" -- because it looks for a body variant the message does
not carry, while the same message holds a 14,996-byte body in the other.
Headers are indexed as text, not just metadata: who sent it and when is much
of what an inbox is searched for. Fields are read defensively, since .msg also
stores contacts and calendar items that have no sender or body.

Tests author genuine OLE2 containers (tests/support/cfb_writer.py) rather than
mocking the format, since a mock of a binary parser is the kind of test that
passes while the parser returns nothing. olefile reads but cannot write CFB.

LibreOffice is added to the image (Writer and Calc only, not the metapackage).
Where the binary is absent the affected types are simply not claimed, so the
registry reports "no processor for type" once instead of failing per document.
Stack created with GitHub Stacks CLIGive Feedback 💬


This PR was generated with the help of AI, and reviewed by a Human


This PR was generated with the help of AI, and reviewed by a Human

@claude

claude Bot commented Aug 6, 2026

Copy link
Copy Markdown

Claude finished @cbcoutinho's task in 3m 25s —— View job


Review

  • Gather context (PR description, diff, prior review rounds)
  • Read changed files
  • Review code quality, bugs, security, performance, test coverage
  • Post final review

This is round 4 of review on this PR; rounds 1–3 already covered the substantive findings (settings-driven timeout, rendition size cap, .xls unit coverage, code-page tests, docstring fixes) and they're all landed. I read every changed file independently rather than re-reviewing the same ground — summary below.

Independently verified, not just re-read:

  • Dependency claim checks out. uv.lock only gains olefile, openpyxl, and openpyxl's own et-xmlfile transitive — no extract-msg/red-black-tree-mod sneaks in, confirming the docker:S8541 rationale in _msg_reader.py's docstring.
  • MAPI property tags in _msg_reader.py are correct (0037=PR_SUBJECT, 5D01=PR_SENDER_SMTP_ADDRESS, 0E04/0E03=To/Cc, 1000=PR_BODY, 3707/3704=attachment names, 0x0039/0x0E06=submit/delivery SYSTIME, 0x3FDE=PR_INTERNET_CPID) — checked each against MS-OXPROPS from memory and they all line up. The <I unpack of the fixed-property entry correctly reconstructs tag<<16 | type from the little-endian layout.
  • ProcessorRegistry.find_processor re-reads supported_mime_types per call (not cached at registration), so SpreadsheetProcessor's LIBREOFFICE_AVAILABLE-gated MIME set in spreadsheet.py:53-61 behaves correctly even though SpreadsheetProcessor is registered unconditionally in __init__.py:53 regardless of LibreOffice's presence.
  • Ran the intended reasoning on the size-cap plumbing (office.py:97-115) and the process-limiter nesting (_libreoffice.py:104-121) — both are correct: the limiter is acquired/released once per conversion, before the delegated PDF parse acquires it in turn, so they never nest; the rendition oversize check reuses oversize_result_for_size rather than restating the threshold.

Minor, non-blocking observations:

  • office.py's oversize path reuses oversize_result_for_size, whose log message is hardcoded "PDF %s is..." (registry.py:636) — for a .docx source this logs "PDF contract.docx is 12.0 MB", which reads a little oddly since the file isn't a PDF. Cosmetic only (it's shared code, not new in this PR), not worth a special case.
  • The code-page fallback (_msg_reader.py:171-175) resolves an Internet Codepage number to a Python codec via the literal string f"cp{codepage}". That covers the common Windows code pages (1252, 1251, ...) but wouldn't resolve some declared codepages (e.g. 65001/UTF-8, whose canonical Python alias is cp65001 — which is registered by CPython's aliases table, so this one happens to work, but the general pattern is fragile for codepages without a cp<N> alias). It's already defensive — an unmapped value falls back to cp1252 rather than raising, and that fallback path is explicitly tested (test_an_unknown_code_page_falls_back_instead_of_raising) — so this is a "good enough, worth knowing" note rather than a bug.
  • Both previously-identified follow-ups (asymmetric size re-check on the .xls→.xlsx path, no rlimit isolation on openpyxl/OLE parsing) are correctly deferred to Deck chore(deps): update nextcloud-33 docker tag to v33.0.6 #987 rather than scope-crept into this PR — agree with that call given the size of this diff already.

Test coverage is genuinely strong: real OLE2 containers via tests/support/cfb_writer.py rather than mocking the binary format (so the .msg reader is tested against bytes it would actually receive), mocked unit coverage for every LibreOffice-dependent branch (so CI without soffice still exercises the .xls and rendition-size-cap logic), and integration tests that render real fixtures through the real binary and assert the specific defect (merged-cell column shift) the rendition route exists to avoid.

No blocking issues found. Nice diff — the per-format "read it the way that measured best" reasoning is well-argued and the docstrings carry real information (numbers, not vibes).

@cbcoutinho
cbcoutinho force-pushed the feat/office-msg-processors branch from d3163fb to d4eeb57 Compare August 6, 2026 09:17
cbcoutinho added a commit that referenced this pull request Aug 6, 2026
…unit

Review follow-up on #1265.

The `msg.py` module docstring still claimed the processor was "read with
extract-msg" and cited extract-msg's numbers as the reason, left over from
before the switch to the olefile reader -- while `_msg_reader.py`'s own
docstring told the real story and extract-msg is not even a dependency. It now
describes what the module actually does and points at the reader for the
rationale, keeping the measured comparison but attributing it correctly.

Rename `_libreoffice.convert`'s `timeout` to `timeout_seconds`, matching
`document_parse_timeout_seconds` and its siblings, and document why the cap is
a parameter at all rather than the caller's own `fail_after`: expiry is
translated here into `LibreOfficeError`, the single exception type both callers
already handle (python:S7483).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cbcoutinho
cbcoutinho force-pushed the feat/office-msg-processors branch from 1c68b00 to f260d26 Compare August 6, 2026 17:33
@cbcoutinho

Copy link
Copy Markdown
Owner Author

Round-1 findings addressed in f260d26f:

  • 🔴 Stale msg.py module docstring: correct catch — it was left over from before the switch to the olefile reader, and actively misleading. Rewritten to describe what the module does (format the reader's fields) and to point at _msg_reader for the rationale, keeping the measured comparison but attributing it to the right implementation.
  • Sonar python:S7483 (not in your review, but on the same lines): renamed convert(timeout=…) to timeout_seconds, matching document_parse_timeout_seconds and its siblings, and documented why the cap is a parameter rather than the caller's own fail_after — expiry is translated here into LibreOfficeError, the one exception type both callers already handle.

Left as-is, with reasons:

  • Single progress_callback before the conversion: a mid-conversion signal would need a poller like unstructured.py's, which is more machinery than the stall warrants; the conversion is one await and the timeout already bounds it.
  • _sheet_rows dropping interior blank rows: deliberate. The goal is retrieval, not visual fidelity, and no cell content is lost — the docstring says so.
  • No rlimit/timeout isolation on openpyxl / the OLE parse: agreed this is worth covering, and agreed it is not a regression (no non-PDF processor has it). Tracked as follow-up on Deck chore(deps): update nextcloud-33 docker tag to v33.0.6 #987 rather than widened into this PR — a zip-bomb .xlsx is a real threat model for a multi-tenant deployment and deserves its own change, since the pre-download size cap bounds compressed bytes only.

3374 unit tests pass, 5 office integration tests against real LibreOffice pass; ruff/ty green.

cbcoutinho added a commit that referenced this pull request Aug 6, 2026
…n size

Round-2 review follow-up on #1265.

The LibreOffice timeout was hardcoded at 120s with no override, unlike every
comparable timeout here (`document_parse_timeout_seconds`, `docling_timeout`,
`unstructured_timeout`). Adds `document_office_timeout_seconds` and threads it
through registration for both processors that convert.

A rendition reached the PDF engine directly rather than back through
ProcessorRegistry, so it skipped the size cap an uploaded PDF has to pass. The
source is already capped before download, but rendering is not size-preserving
-- a modest .doc of dense vector figures can render far larger -- so the cap is
now applied to the bytes actually about to be parsed, reusing
`oversize_result_for_size` rather than restating the threshold.

Also from the review: say why `extract_images=False` is forced for renditions;
note in `_msg_reader` that only the plain-text body property is read, and what
it would take to add the HTML/RTF variants; cover the legacy `.xls` branch with
mocked unit tests so it is not exercised only by the LibreOffice-gated
integration test; and cover the code-page decode path and its unknown-CPID
fallback, which the `_build_msg` fixture could already produce but no test used.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cbcoutinho added a commit that referenced this pull request Aug 6, 2026
…n size

Round-2 review follow-up on #1265.

The LibreOffice timeout was hardcoded at 120s with no override, unlike every
comparable timeout here (`document_parse_timeout_seconds`, `docling_timeout`,
`unstructured_timeout`). Adds `document_office_timeout_seconds` and threads it
through registration for both processors that convert.

A rendition reached the PDF engine directly rather than back through
ProcessorRegistry, so it skipped the size cap an uploaded PDF has to pass. The
source is already capped before download, but rendering is not size-preserving
-- a modest .doc of dense vector figures can render far larger -- so the cap is
now applied to the bytes actually about to be parsed, reusing
`oversize_result_for_size` rather than restating the threshold.

Also from the review: say why `extract_images=False` is forced for renditions;
note in `_msg_reader` that only the plain-text body property is read, and what
it would take to add the HTML/RTF variants; cover the legacy `.xls` branch with
mocked unit tests so it is not exercised only by the LibreOffice-gated
integration test; and cover the code-page decode path and its unknown-CPID
fallback, which the `_build_msg` fixture could already produce but no test used.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cbcoutinho
cbcoutinho force-pushed the feat/office-msg-processors branch from e2a58e9 to be60182 Compare August 6, 2026 17:56
@cbcoutinho

Copy link
Copy Markdown
Owner Author

Round-2 findings addressed in be601822:

  • 🟡 Conversion timeout had no settings knob: added DOCUMENT_OFFICE_TIMEOUT_SECONDS and threaded it through registration for both converting processors. You were right that it was the odd one out — documented alongside its siblings in docs/configuration.md.
  • 🟡 Oversize guard bypassed for renditions: fixed rather than accepted. The source is capped before download, but as you say rendering is not size-preserving, so the cap is now applied again to the rendition bytes about to be parsed, reusing oversize_result_for_size rather than restating the threshold. Two tests: an oversize rendition is rejected without the PDF parse being reached, and one within the cap still parses.
  • 🟢 Unexplained extract_images=False: commented. The images in a rendition are LibreOffice's raster of the source's own figures, one indirection from anything a user could be shown, and they would be written to disk only to be discarded with the rendition.
  • 🟢 .xls branch only covered by the LibreOffice-gated integration test: added three mocked unit tests — converts to xlsx (never pdf), conversion failure maps to ProcessorError, and .xlsx never touches LibreOffice at all.
  • 🟢 Code-page path untested: added both cases, a declared cp1252 decode and an unknown CPID falling back rather than raising.
  • 🟢 Only the plain-text body property is read: noted in _msg_reader's docstring, including which properties the HTML/RTF variants live in and that 1009 is the point where compressed-rtf becomes cheaper than more code here.

Also documented the two-caps-and-a-slot interaction in docs/configuration.md, since a rendition now pays the office timeout, the size cap twice, and a parse slot.

3391 unit tests pass, 5 office integration tests against real LibreOffice pass; ruff/ty green.

cbcoutinho added a commit that referenced this pull request Aug 7, 2026
…unit

Review follow-up on #1265.

The `msg.py` module docstring still claimed the processor was "read with
extract-msg" and cited extract-msg's numbers as the reason, left over from
before the switch to the olefile reader -- while `_msg_reader.py`'s own
docstring told the real story and extract-msg is not even a dependency. It now
describes what the module actually does and points at the reader for the
rationale, keeping the measured comparison but attributing it correctly.

Rename `_libreoffice.convert`'s `timeout` to `timeout_seconds`, matching
`document_parse_timeout_seconds` and its siblings, and document why the cap is
a parameter at all rather than the caller's own `fail_after`: expiry is
translated here into `LibreOfficeError`, the single exception type both callers
already handle (python:S7483).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cbcoutinho added a commit that referenced this pull request Aug 7, 2026
…n size

Round-2 review follow-up on #1265.

The LibreOffice timeout was hardcoded at 120s with no override, unlike every
comparable timeout here (`document_parse_timeout_seconds`, `docling_timeout`,
`unstructured_timeout`). Adds `document_office_timeout_seconds` and threads it
through registration for both processors that convert.

A rendition reached the PDF engine directly rather than back through
ProcessorRegistry, so it skipped the size cap an uploaded PDF has to pass. The
source is already capped before download, but rendering is not size-preserving
-- a modest .doc of dense vector figures can render far larger -- so the cap is
now applied to the bytes actually about to be parsed, reusing
`oversize_result_for_size` rather than restating the threshold.

Also from the review: say why `extract_images=False` is forced for renditions;
note in `_msg_reader` that only the plain-text body property is read, and what
it would take to add the HTML/RTF variants; cover the legacy `.xls` branch with
mocked unit tests so it is not exercised only by the LibreOffice-gated
integration test; and cover the code-page decode path and its unknown-CPID
fallback, which the `_build_msg` fixture could already produce but no test used.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cbcoutinho
cbcoutinho force-pushed the feat/office-msg-processors branch from be60182 to b1d8700 Compare August 7, 2026 01:32
@cbcoutinho

Copy link
Copy Markdown
Owner Author

Round-3: no code changes — the two substantive items were the round-2 pair (settings-driven conversion timeout, rendition size cap), both already in b1d8700d.

On your three observations:

  • Asymmetric size re-check between the conversion paths — agreed, and noted on Deck chore(deps): update nextcloud-33 docker tag to v33.0.6 #987 as you suggested rather than fixed here. The .xls → .xlsx path has no size check on the converted bytes before openpyxl reads them; as you say the inflation risk is much lower for container-to-container than for rendering, but it is the same category as the zip-bomb item already on that card. The note lists all three inputs the eventual isolation fix should cover: an uploaded .xlsx, a LibreOffice-produced one, and the OLE parse in _msg_reader.
  • health_check() asymmetry (import-probe vs binary-probe) — left as-is deliberately. OfficeDocumentProcessor probes soffice because its dependency is an external binary that can genuinely be absent from an image; msg/spreadsheet depend on wheels that are installed or the module would not import. Making them symmetric would mean either a pointless round-trip parse or dropping a real check.
  • No behavioural issues found — noted, thank you for reading the MAPI/FILETIME/escaping paths closely.

Rebased onto the updated lower PRs (#1263's helper move, #1264's scope fix). 3374 unit tests pass; office integration tests green against real LibreOffice.

cbcoutinho added a commit that referenced this pull request Aug 7, 2026
…unit

Review follow-up on #1265.

The `msg.py` module docstring still claimed the processor was "read with
extract-msg" and cited extract-msg's numbers as the reason, left over from
before the switch to the olefile reader -- while `_msg_reader.py`'s own
docstring told the real story and extract-msg is not even a dependency. It now
describes what the module actually does and points at the reader for the
rationale, keeping the measured comparison but attributing it correctly.

Rename `_libreoffice.convert`'s `timeout` to `timeout_seconds`, matching
`document_parse_timeout_seconds` and its siblings, and document why the cap is
a parameter at all rather than the caller's own `fail_after`: expiry is
translated here into `LibreOfficeError`, the single exception type both callers
already handle (python:S7483).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cbcoutinho added a commit that referenced this pull request Aug 7, 2026
…n size

Round-2 review follow-up on #1265.

The LibreOffice timeout was hardcoded at 120s with no override, unlike every
comparable timeout here (`document_parse_timeout_seconds`, `docling_timeout`,
`unstructured_timeout`). Adds `document_office_timeout_seconds` and threads it
through registration for both processors that convert.

A rendition reached the PDF engine directly rather than back through
ProcessorRegistry, so it skipped the size cap an uploaded PDF has to pass. The
source is already capped before download, but rendering is not size-preserving
-- a modest .doc of dense vector figures can render far larger -- so the cap is
now applied to the bytes actually about to be parsed, reusing
`oversize_result_for_size` rather than restating the threshold.

Also from the review: say why `extract_images=False` is forced for renditions;
note in `_msg_reader` that only the plain-text body property is read, and what
it would take to add the HTML/RTF variants; cover the legacy `.xls` branch with
mocked unit tests so it is not exercised only by the LibreOffice-gated
integration test; and cover the code-page decode path and its unknown-CPID
fallback, which the `_build_msg` fixture could already produce but no test used.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cbcoutinho
cbcoutinho force-pushed the feat/office-msg-processors branch from b1d8700 to 8814e69 Compare August 7, 2026 01:40
cbcoutinho added a commit that referenced this pull request Aug 7, 2026
…unit

Review follow-up on #1265.

The `msg.py` module docstring still claimed the processor was "read with
extract-msg" and cited extract-msg's numbers as the reason, left over from
before the switch to the olefile reader -- while `_msg_reader.py`'s own
docstring told the real story and extract-msg is not even a dependency. It now
describes what the module actually does and points at the reader for the
rationale, keeping the measured comparison but attributing it correctly.

Rename `_libreoffice.convert`'s `timeout` to `timeout_seconds`, matching
`document_parse_timeout_seconds` and its siblings, and document why the cap is
a parameter at all rather than the caller's own `fail_after`: expiry is
translated here into `LibreOfficeError`, the single exception type both callers
already handle (python:S7483).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cbcoutinho added a commit that referenced this pull request Aug 7, 2026
…n size

Round-2 review follow-up on #1265.

The LibreOffice timeout was hardcoded at 120s with no override, unlike every
comparable timeout here (`document_parse_timeout_seconds`, `docling_timeout`,
`unstructured_timeout`). Adds `document_office_timeout_seconds` and threads it
through registration for both processors that convert.

A rendition reached the PDF engine directly rather than back through
ProcessorRegistry, so it skipped the size cap an uploaded PDF has to pass. The
source is already capped before download, but rendering is not size-preserving
-- a modest .doc of dense vector figures can render far larger -- so the cap is
now applied to the bytes actually about to be parsed, reusing
`oversize_result_for_size` rather than restating the threshold.

Also from the review: say why `extract_images=False` is forced for renditions;
note in `_msg_reader` that only the plain-text body property is read, and what
it would take to add the HTML/RTF variants; cover the legacy `.xls` branch with
mocked unit tests so it is not exercised only by the LibreOffice-gated
integration test; and cover the code-page decode path and its unknown-CPID
fallback, which the `_build_msg` fixture could already produce but no test used.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cbcoutinho
cbcoutinho force-pushed the feat/office-msg-processors branch from 8814e69 to 63392b0 Compare August 7, 2026 01:55
cbcoutinho added a commit that referenced this pull request Aug 7, 2026
Round-4 review follow-up on #1265.

The setting was explicitly modelled on `document_parse_timeout_seconds`, but
only the model got a `Validator(..., gte=1)`. A 0 or negative value therefore
reached `anyio.fail_after` in `_libreoffice.convert` and expired every
conversion the instant it started -- so the misconfiguration presented as
"every .doc/.docx fails to parse", with nothing pointing at the setting that
caused it, where its two siblings fail fast at startup and name themselves.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cbcoutinho and others added 4 commits August 7, 2026 04:10
Adds three processors, each reading its format the way that measured best
rather than forcing one route on all of them.

Word documents (.doc/.docx) go through a LibreOffice PDF rendition parsed at
the structured tier. Legacy .doc has no working pure-Python reader at all --
markitdown raises UnsupportedFormatException and docling-serve returns
status=failure -- and for .docx the rendition is *more* faithful than a direct
parse, not less: 98.3% token recall with 22 table rows against mammoth's 17,
because mammoth drops a vertically-merged cell and shifts the rest of that row
one column left, landing answers under the wrong heading. The rendition is a
real PDF, so page numbers and highlight geometry come from the existing path
instead of a second implementation. Hyperlink targets are lost in exchange.

The tier is forced, not classified: the classifier scores a rendition on text
quality and returns tier='fast', and the fast tier extracts ZERO tables from
one, because rendered borders are vector line-art that only find_tables sees.

Spreadsheets (.xls/.xlsx) are read cell-by-cell and deliberately never
rendered. PDF rendering paginated a real workbook into 25 mixed-orientation
pages and recalled only 63.8% of the tokens a direct read recovers -- 567
cells against 2013 -- losing the question numbering entirely, because
LibreOffice honours the print layout. Legacy .xls converts container-to-
container via xlsx, which changes the format without re-laying it out.
Chunks carry `sheet_boundaries` (sheet + cell range) where a PDF carries
pages; a spreadsheet has no geometry to highlight.

Outlook messages (.msg) are read with a small olefile-based reader rather than
`extract-msg`, which depends on `red-black-tree-mod` -- published as an sdist
only, so it cannot satisfy the image's `uv sync --no-build` (no dependency's
setup.py may execute at build time, docker:S8541). The reader is ~80 lines and
recovers slightly more than extract-msg did on a real 1.3 MB thread (15,313
characters against 15,303) about seven times faster, while dropping nine
transitive dependencies. Reading *both* string variants is the substance of
it: markitdown emits 27 bytes for that message -- the literal "# Email
Message\n\n## Content" -- because it looks for a body variant the message does
not carry, while the same message holds a 14,996-byte body in the other.
Headers are indexed as text, not just metadata: who sent it and when is much
of what an inbox is searched for. Fields are read defensively, since .msg also
stores contacts and calendar items that have no sender or body.

Tests author genuine OLE2 containers (tests/support/cfb_writer.py) rather than
mocking the format, since a mock of a binary parser is the kind of test that
passes while the parser returns nothing. olefile reads but cannot write CFB.

LibreOffice is added to the image (Writer and Calc only, not the metapackage).
Where the binary is absent the affected types are simply not claimed, so the
registry reports "no processor for type" once instead of failing per document.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A soffice process is the heaviest thing the ingest path spawns -- it holds the
source document and the rendered output at once -- and nothing capped how many
could run. Indexing a folder of .doc files would start one per concurrent task
and exhaust the pod's memory, which is precisely what the existing parse-slot
limiter exists to prevent for the lighter parse workers.

Acquire that same limiter (`document_parse_process_slots`) around the
conversion. It is taken and released before the delegated PDF parse acquires
it in turn, so the two never nest, and one setting now bounds every heavy
document subprocess rather than only half of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…unit

Review follow-up on #1265.

The `msg.py` module docstring still claimed the processor was "read with
extract-msg" and cited extract-msg's numbers as the reason, left over from
before the switch to the olefile reader -- while `_msg_reader.py`'s own
docstring told the real story and extract-msg is not even a dependency. It now
describes what the module actually does and points at the reader for the
rationale, keeping the measured comparison but attributing it correctly.

Rename `_libreoffice.convert`'s `timeout` to `timeout_seconds`, matching
`document_parse_timeout_seconds` and its siblings, and document why the cap is
a parameter at all rather than the caller's own `fail_after`: expiry is
translated here into `LibreOfficeError`, the single exception type both callers
already handle (python:S7483).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…n size

Round-2 review follow-up on #1265.

The LibreOffice timeout was hardcoded at 120s with no override, unlike every
comparable timeout here (`document_parse_timeout_seconds`, `docling_timeout`,
`unstructured_timeout`). Adds `document_office_timeout_seconds` and threads it
through registration for both processors that convert.

A rendition reached the PDF engine directly rather than back through
ProcessorRegistry, so it skipped the size cap an uploaded PDF has to pass. The
source is already capped before download, but rendering is not size-preserving
-- a modest .doc of dense vector figures can render far larger -- so the cap is
now applied to the bytes actually about to be parsed, reusing
`oversize_result_for_size` rather than restating the threshold.

Also from the review: say why `extract_images=False` is forced for renditions;
note in `_msg_reader` that only the plain-text body property is read, and what
it would take to add the HTML/RTF variants; cover the legacy `.xls` branch with
mocked unit tests so it is not exercised only by the LibreOffice-gated
integration test; and cover the code-page decode path and its unknown-CPID
fallback, which the `_build_msg` fixture could already produce but no test used.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Round-4 review follow-up on #1265.

The setting was explicitly modelled on `document_parse_timeout_seconds`, but
only the model got a `Validator(..., gte=1)`. A 0 or negative value therefore
reached `anyio.fail_after` in `_libreoffice.convert` and expired every
conversion the instant it started -- so the misconfiguration presented as
"every .doc/.docx fails to parse", with nothing pointing at the setting that
caused it, where its two siblings fail fast at startup and name themselves.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cbcoutinho
cbcoutinho force-pushed the feat/office-msg-processors branch from eecfe1f to 2011233 Compare August 7, 2026 02:10
@sonarqubecloud

sonarqubecloud Bot commented Aug 7, 2026

Copy link
Copy Markdown

@cbcoutinho

Copy link
Copy Markdown
Owner Author

Round-4 finding addressed in eecfe1f6 (now 2011233f after restacking).

  • 🟡 DOCUMENT_OFFICE_TIMEOUT_SECONDS had no startup validator: correct, and the inconsistency is exactly as you frame it — I said in round 1 that the setting was modelled on document_parse_timeout_seconds, then gave it the model's shape without the model's guard. A 0 or negative value reached anyio.fail_after and expired every conversion immediately, so the misconfiguration presented as "every .doc/.docx fails to parse" with nothing naming the setting responsible.

    Added Validator("DOCUMENT_OFFICE_TIMEOUT_SECONDS", gte=1) alongside its two siblings, with tests for rejection at 0 and -1, acceptance of a positive value, and the default when unset.

3392 unit tests pass; ruff/ty green; Sonar gate OK.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant